Modern software applications and systems can be extremely complex, and may contain thousands or millions of lines of interrelated code spread across numerous files. Modern programs are also more interdependent than ever on other programs, as internet-based and other networked or otherwise connected systems continue to supplant stand-alone applications. Such networked applications may depend on many other programs to pass them appropriate data in order to run properly and without errors. Complex software systems carry with them a great risk of errors, such as so-called bugs.
Software generally goes through a number of iterative revisions as it moves from conception to initial launch (e.g., in alpha or beta test), and then through to commercial release. The process of identifying and tracking problems in software code is generally referenced as quality assurance (QA), and entire departments may be devoted to such a function. One way that QA engineers attempt to identify problems in software for later eradication is known as a “test case.” A test case is a set of conditions or variables under which a tester (whether human or machine) may determine whether a requirement for the operation of a piece of software is satisfied. A written test case can include a description of the functionality to be tested (taken, e.g., from the requirements for the software, or from use cases, such as specific examples of how the software is intended to be used), and the expected output of the case, so that the functionality can be verified as working correctly. As a simple example, an engineer may test a piece of software for adding two numbers together by running the software using two exemplary numbers, and checking whether the output matches the output that is expected. As a more realistic example, an engineer may populate a database that is to be accessed by a program with information, and then may query the database using the program to see if the appropriate data is returned and presented.
An engineer often sets a known input for the software and determines an expected output before running the software on the test case. For example, for software to add two numbers, the engineer may choose to pass 4 and 6 to the program as arguments. If the software outputs 10, then the requirement for the test case will be satisfied. In more complex software, it may take many test cases to determine whether a requirement is actually satisfied. In addition, test cases may have to be re-run as software is modified from early versions through later versions, e.g., to add functionality or to eliminate bugs. Each version of compiled and tested software is often referenced as a “build” of the software. Test cases generally need to be run for every build. A collection of test cases is often referenced as a test suite.
Generally, QA engineers can use a test case management system during testing to help them plan testing, run the tests or test cases, and report the results of the testing. For the plan, the system may create trackable and loggable test cases to test one or more functionalities of an application, and associate test cases with a particular bug or feature. For the run, the system may determine a sequence of test cases that need to be run, and may run through the test cases and log the results in a database. For the reporting, the system may find information about test cases that were run on a specific “build” of the software, and may track progress of software during the QA cycle. The reporting may also report on code coverage, along with load and performance statistics and test progress, among other things.